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ABSTRACT 



This paper is concerned with the assessability of 
metacognition, asserting that unless metacognition can be assessed, then it 
will never be considered a legitimate part of curriculum. The paper focuses 
on a pilot study conducted with grade- six students in the curriculum domain 
of mathematics. It reports the results of student questionnaires and 
interviews that asked students to describe what they did when they solved 
open-ended mathematics questions. Questionnaires were completed by 15 sixth 
graders from 2 schools, and 5 students participated in interviews. The pilot 
study was planned to highlight key aspects of students' metacognitive 
thinking, but it also raised important methodological questions about the 
validity of assessments of student thinking. Existing definitions of 
metacognition appear to be imprecise and difficult to interpret in 
operational terms. However, self-reporting appears to be a valid method for 
researching metacognition when used in conjunction with other reliable 
methods and checking procedures . An appendix presents a table of 
metacognitive behavior. (Contains 2 figures, 1 table, and 68 references.) 
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Abstract 



The importance of metacognition for student learning has been widely acknowledged 
(Anderson and Walker, 1991, Biggs, 1987, Birenbaum, 1996, Brown and De Loache, 1983, 
Pintrich, and De Groot, 1990 Wilson and Wing Jan, 1993, Wilson and Wing Jan, in press). 
But the practicalities associated with teaching for metacognition and monitoring metacognition 
are not clear. This paper is concerned about'the assessability of metacognition. It is asserted 
that unless metacognition can be assessed then it will never be considered a legitimate part of 
curriculum. 



This paper focuses on the results of a pilot study that has been conducted with upper primary 
students in the curriculum domain of mathematics. It reports the results of student 
questionnaires and interviews which asked students to describe what they did when they solved 
open ended mathematics problems. The pilot study was planned to highlight key aspects of 
students' metacognitive thinking but it also raised important methodological questions about the 
validity of assessments of student thinking. 
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BEYOND THE BASICS: ASSESSING STUDENTS' METACOGNITION 



Jeni Wilson 

Department of Learning, Assessment and Special Education. Faculty of Education, 
University of Melbourne. 

Paper Presented at the 14th Annual Hong Kong Educational Research Association 
Conference. Hong Kong. November, 1997. 

Introduction 

Many curriculum documents in Australia promote the importance of metacognition, for 
example, The Australian National Statement on Mathematics, Australian Educational Council. 
(1991) and Ministry of Education, (1989). Many teachers and researchers believe that students' 
thinking about their own thinking and learning will improve the educational outcomes of 
students (Anderson and Walker, 1991, Biggs, 1987, Perry, 1989, Stacey, 1990, Wittrock, 
1986). Students who exhibit advanced skills in a content area are assumed to have well 
developed metacognitive skills in that area (Royer, Cisero and Carlo, 1993). But there is much 
confusion about the meaning of metacognition. The term 'metacognition' has not been clearly 
defined, little teacher reference material about metacognition has been written and even less 
assessment material is available to monitor metacognition. With little teacher support, an 
'overcrowded curriculum’ (Pigdon and Woolley, Eds, 1992) and a move to test the basics, 
metacognition may be seen as unachievable as a curricula goal. 

Projects such as PEEL (Baird and Northfield, Eds, 1992) embody the notion that 
metacognition is important and that it can be taught. Clarke (1988) argues that assessment 
should monitor the development of the attributes and capabilities that we value. This paper 
asserts that if metacognition can be taught it should be assessed. Assessment of any aspect of 
the curriculum increases the likelihood of it being taught. If metacognition is desirable then 
tasks which monitor metacognition would be worthwhile identifying. If metacognition cannot 
be assessed then it will remain attractive at a theoretical level only. It is sometimes argued that 
assessment should not lead curriculum (Royer, Cisero and Carlo, 1993) but in reality this is 
often how change is instigated. This is particularly true of high stakes assessment (Clarke and 
Stephens, 1996). 

Some writers make the link between assessment and metacognition by suggesting that student 
self-assessment, where students are involved in the process of making judgements about their 
own learning (Boud, 1991), is a means to develop metacognition. As an instructional tool self- 
assessment could be used to develop metacognitive skills. As an assessment tool self- 
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assessment could be used to assess cognition. The assessment of metacognition could lie in the 
judgements about the effectiveness of student use of the self-assessment medium or self- 
assessment items could ask students to reflect on their own thinking. 

The development of 'teacher friendly' assessment materials in the area of metacognition would 
improve the likelihood of metacognition being seen as important, viable and basic to student 
development. This study was the initial stage of a larger, overarching project which aims to 
identify strategies for assessing metacognition. This paper reports on the results of the pilot 
study. It highlights methodological questions about instruments used to assess student 
metacognitive thinking. 

Defining assessment 

The terms assessment and evaluation are used differently by different people and often 
interchangeably. Sims (1992) reports that the effect of the terms assessment, evaluation and 
measurement being used synonymously has caused confusion and in some cases mistrust and 
hostility. 

The Victorian Ministry of Education in 1988 distinguished between assessment and 
evaluation. They defined assessment as a ‘process concerned with gathering information 
about students’ competencies where the focus is not only what has been achieved by students 
but also how they have gone about learning.’ (1988 : 96) They define evaluation as the 
process ‘where judgments are made about the worth of educational programs.’ (1988 : 96) 
Wilson also explores the connection between these two processes: ‘The term assessment is 
given to the process of collecting and analysing information about student 
performance. ...Evaluation depends on assessment because it makes use of this collected 
information to make informed decisions about future activities and programs. ’(1992 : 1 10- 
111). Later, Wilson and Fehring (1995) use the term evaluation to denote the making of 
judgments about students’ work and the effectiveness of teaching and learning methods, 
programs and resources. 

The term assessment will be used throughout this paper to refer to the collection of information 
about students ’ learning. 

Defining metacognition 

Despite a growing interest in the concept of metacognition over the last twenty years and 
attempts to define reflective and metacognitive thinking, a lack of clarity has existed (Brown. 
1987, Munroe, 1993, Weinert, 1987). Metacognition is defined below after a brief review of 
the literature. 
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The literature associated with metacognition dates back to Dewey in 1933. Although he didn't 
use the word metacognition, Dewey spoke of 'reflective self-awareness’, -the importance of 
active, persistent and careful consideration of beliefs and knowledge. Flavell (1976) was the 
first to use the term 'metacognition', which refers to the individual's awareness, consideration 
and control of his or her own cognitive processes and strategies. Since then there have been a 
variety of meanings given to the term. 

Perry (1989) summarised theorists' attempts to classify and describe metacognition. She refers 
to Brown and De Loache (1978) who speak of knowing about knowing, Kontos (1983) who 
describes knowing and conscious control of cognitive processes, Brown, Bransford, Ferrara, 
and Campione, (1983) who state that knowledge about cognition and self-regulation of 
cognition are equally important twin aspects of metacognition, and Paris, Saarnio, and Cross, 
(1984) who simply define metacognition as what you know about how you know. Brown 
defined metacognition simply as referring to 'one's knowledge and control of one's own 
cognitive system.' (1987 : 67). Schraw and Dennison (1994) include a reflective component in 
their definition of metacognition: 'Metacognition refers to the ability to reflect upon, understand 
and control one's learning.' (1994: 460) 

Garofalo and Lester (1984) claim the confusion over the term metacognition is due to 
metacognition having two separate but related aspects: knowledge and beliefs about cognition 
and the regulation and control of cognitive actions. They suggest that it is difficult to 
distinguish cognition from metacognition. Garofalo and Lester explain the difference and 
relationship between them by simplifying cognition as ‘involved in doing, whereas 
metacognition is involved in choosing and planning what to do and monitoring what is being 
done.’ (1984 : 163) Metacognition is knowing when, where and how to use knowledge and 
beliefs. 

Brown (1987) and Schoenfeld (1990) agree that an understanding of the term metacognition is 
difficult because it is used to refer to two distinct areas of research: knowledge about cognition 
and regulation of cognition. The concept of metacognition is difficult to use because it has 
'multiple and almost disjoint meanings, for example, knowledge about one's thought processes 
and self-regulation during problem solving’. (Schoenfeld, 1990: 1) 

Weinert (1987) suggests that the problems of definition are made apparent when attempting to 
apply the term to specific instances. Questions are raised about whether metacognitive 
knowledge must be utilised, whether it must be conscious and verbalisable and whether it must 
be generalized across situations. The problem of definition has implications for methodology 
because without a clear definition the investigation cannot proceed. A definition provides the 
parameters for research and a means to analyse the results of the investigation. 
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My synthesis of the preceding literature overview takes the form of the following definition: 
Metacognition refers to the awareness individuals have of their own thinking and their ability 
to evaluate and regulate their own thinking. 



The model below represents metacognition as it is employed in this paper. It is used as a 
framework. Borkowski and Muthukrishna (1992) stress the importance of developing working 
models for metacognition in curriculum delivery: 'A working model provides a scheme for 
organising knowledge, a framework in which to incorporate new information, and a 
springboard for launching future actions.' (1992 : 480). The model shows that there are three 
functions of metacognition which include: Awareness, Evaluation and Regulation of one's own 
thinking. Awareness and Evaluation are components of the thinking activity I will call 
Monitoring. Reflection is the mediating process whereby Awareness may become Evaluation 
and Evaluation may be transformed into Regulation of the thinking processes. (See Figure 1) 



Metacognitive 

Functions 



METACOGNITION 




Objects 

of 

Reflection 



Self 


Task/Context 


Thinking about own: 


Thinking about own: 


* Learning 


* Choice of 


* Thinking 


strategy 


* Behaviour 


* Use of Strategy 


* Abilities 

* Progress 


* Use of tools 



Self 


Task/Context 


Thinking about own: 

* Learning 

* Thinking 

* Behaviour 


Thinking about own: 

* Choice of 
strategy 

* Use of Strategy 

* Use of tools 



Figure 1. Model of Metacognition 



The three functions of metacognition are clarified below: 



Metacognitive Awareness relates to an individual’s awareness of where they are in the learning 
process, their knowledge about personal learning strategies and what needs to be done (Baird, 
1991). 
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Metacognitive Evaluation refers to judgements made regarding one's thinking capacities and 
limitations as these are employed in a particular situation or as self-attributes. For example, 
individuals could be making a judgement on the effectiveness of their thinking and strategy 
choice. 

Metacognitive Regulation occurs when individuals modify their thinking. They make use of 
their metacognitive skills to control their knowledge and thinking. They reflect on their 
knowledge about self and strategies (how and why they may use particular strategies). Schraw 
and Dennison (1994) refer to these ways of thinking about thinking as declarative, procedural 
and conditional knowledge. These include abilities to plan, self-correct, set goals and the 
effective use of one's own cognitive resources (Brown, 1978, Scardamalia, Bereiter and 
Steinbach, 1985). Metacognitive skills are needed to make use of knowledge in the process of 
regulating thinking. 

Learners who are metacognitively aware- who have developed metacognitive skills and are able 
to access their metacognitive knowledge- have the capacity to monitor and direct their own 
learning. Regulation (which is sometimes referred to as control) is associated with active 
reflection on one's own thinking and involves deliberate decision making about what to do and 
how to act. In other words, when the learner monitors their thinking, makes a judgement about 
their own thinking or consciously decides to act upon their reflections they have been 
metacognitive. 

It is acknowledged that metacognition occurs within a context (for example, a classroom) and 
that other aspects related to the context of learning such as students' prior knowledge, abilities, 
preferred ways of learning, values and expectations (Biggs, 1993) and volition (Corno, 1993) 
affect the nature of learning and thinking. These have not been discounted as important because 
they have not been represented on the model (Figure 1). 

Metacognition and mathematics 

In research studies investigating mathematics understanding and metacognitive decision 
making, Goos (1993, 1994, 1995) makes claims about metacognitive aspects of mathematical 
thinking: 'Metacognitive processes are considered to be an important factor influencing 
problem solving performance.' (Goos, 1993 : 1). Schoenfeld, (1987), Venezky and Bregar. 
(1990) agree that the ability to monitor one's own learning increases the effectiveness of 
problem solving. 

Stacey claims that: ‘Good problem solvers tend to show more meta-cognitive knowledge, ie an 
awareness of the processes of learning and mathematics. ’(1990:6) Research in the field of 
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mathematics and metacognition has reported that students having difficulties with mathematics 
do not use a range of cognitive or metacognitive strategies, Cardelle-Elewar (1992) and Munro 
(1993). These suggestions of a link between metacognition and effective problem solving in 
mathematics strengthen the case for investigating the assessment of metacognition. 

Methodology 



In order to monitor metacognition, specific data collection methods and checking procedures 
were chosen to increase the trustworthiness and reliability of the study. In summary this 
involved: the use of questionnaires (including an inventory for student self-reporting) and 
interviews with grade 6 students in 2 different schools and the use of a questionnaire with one 
teacher. 

The analysis undertaken in this paper relies on inferences made by the researcher about the type 
of metacognitive thinking. The researcher's interpretation should model the metacognitive 
activity of the student. In order for the reader to trust the analysis of the data, the researcher 
must make the analysis methods explicit. In this study the metacognitive behaviors in the 
inventory (part of the student questionnaire) used to gauge levels and make judgements about 
metacognitive performance have been based on behaviors specified on the table on 
metacognitive behavior. This metacognitive behavior table has been based on my definition of 
metacognition and devised after a thorough examination of the literature (See appendix 1). Each 
item has been constructed by adapting the assertions of other authors about metacognitive 
behavior. The items have been listed in the results section. 

Questions of legitimacy 

There are many objections raised about the legitimacy of researching and assessing 
metacognition. For example: Metacognition cannot be assessed because: 

1 . An agreement on a definition has not been achieved. Reseachers cannot assess what you 
cannot define. 

2. Reseachers can only assess the consequences of metacognition. Teachers may interpret 
outcomes as indicative of thinking but performance doesn’t indicate metacognition only 
cognition. 

3. Self-reporting has commonly been used in research about metacognition. But the validity of 
self-reporting is questionable (Nisbett and Wilson, 1977, Nuthall and Alton-Lee, 1995). 
Thinkers may not be conscious of their own thinking and they may not be able to accurately 
describe it. 

These objections are expanded below and addressed in this paper. 
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Garofalo and Lester (1984) suggest there are at least three reasons why the link between 
metacognition and mathematics performance has not been studied systematically. Firstly, they 
claim, mental activity is difficult to observe and analyse. If, as some psychologists suggest, 
people have no direct access to their mental processes, self-reports (a technique often associated 
with metacognitive research) should be considered highly questionable. Secondly, when self- 
reports are used while subjects perform a task this may affect cognitive processes. The 
methodology may create the effect it seeks to study: self-reporting may stimulate metacognition 
rather than provide data on cognition and metacognition. It is interesting, given the connection 
made between assessment and metacognition in this study, that Clarke (1992 and 1996) also 
alerts us to the influence assessment has on student mathematical behavior and performance. 

He reports that it distorts the behavior it was intended to monitor. Indeed most forms of 
research have the potential to distort behavior. 

Webb (1966) also raised the issue about affecting data during the use of some techniques (such 
as questionnaires and interviews). He claimed they intrude into the natural setting and may, in 
the process of measuring, create and reconstruct atypical responses to those which may 
otherwise have occurred. The same may be argued for many other techniques used in a range 
of research. Research into metacognition compounds this problem by employing as the agency 
for data collection the very process about which data is sought. Thirdly, Garofalo and Lester 
(1984) assert the problem of terminology clarification continues to exist. As long as 
metacognition as a phenomena remains ill-defined, Garofalo and Lester claim research in the 
area will attract little attention. Although the literature search results would agree with the lack 
of shared meaning for the term metacognition, the amount of research conducted in the field 
suggests that the issue has created intrigue and engaged many researchers, particularly since 
1984. 

The validity of verbal methods of data collection in the area of thinking processes have been 
questioned. Increased linguistic abilities are sometimes given as a reason as to why student 
metacognitive awareness and control is reported to increase with age. But oral articulation of 
elements of thinking may not represent actual thinking processes. Nisbett and Wilson (1977) 
argue that students are unable to give accurate explanations for their behavior. Thorpe and 
Satterly (1990) share a concern for the adequacy of measures of metacognition. They suggest 
that when measures rely on self-reporting they are also heavily dependent upon linguistic skills 
and therefore results may merely reflect language development rather than metacognitive 
change. Articulation of elements of thinking may not represent actual thinking processes. 

Thorpe and Satterly (1990) also suggest that metacognition is specific to the task from which it 
is derived and question the transfer of such skills. The implication of this statement is the need 
for a thorough and detailed analysis of the use of metacognition in different contexts to ensure 
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that metacognition exists in more than one situation or with different types of problems. The 
use of different tasks is used in this study to cross check information. 



Choice of methods 

The methods used in this study have been carefully chosen after consideration of the following: 

• questions of legitimacy in assessing metacognition, 

• the effects of the various data collection techniques on metacognitive behavior and the 

• suitability of each method when used with children. 

A short questionnaire which included open-ended questions, an inventory of metacognitive 
behaviors, together with an interview to clarify responses formed the basis of student data 
collection with regards to metacognitive thinking. • 

Interviews 

The principle advantage of the interview is that the collection of data is through direct verbal 
interaction. This allows for probing if more complex responses are needed. The procedure 
allows individual responses to be followed up. Details about inconsistencies or questions of the 
interviewer can be collected which are not possible through questionnaires. The data collected 
was compared to the teacher's reported problem solving instructions. 

Interviewing procedures are based on the assumption that 'the person interviewed has insight 
into the cause of her behavior.' Cohen and Manion ( 1994 : 283). Cohen and Manion ( 1994) 
also suggest that this is rarely achieved. I believe the difficulty of respondents' describing their 
own thinking is heightened when dealing with children and when trying to monitor cognition or 
metacognition. Where written responses are difficult, interviews provide a forum for verbal 
responses. 

But the flexibility and personal nature of the interview also allows for subjectivity and bias. 
Borg and Gall (1989) believe that direct interaction through the interview process is both the 
major advantage and disadvantage of interviews. Cohen and Manion (1994) support the idea 
that one way to validate the interview is to compare it with another measure. If the two 
measures agree validity is assumed. They call this comparison 'convergent validity.' ( 1994: 
281) The use of student questionnaires and teacher interviews were used in this study. 

It was decided that 15-20 minutes was optimal to maintain concentration of children in 
interviews. Pines, Novak, Possner, Van Kirk, (1978) suggest that time beyond this limit can 
produce: ' adverse results in both the quality and validity of the data, and in the affective 
response of the subject.' (1978 : 22) 



4 November 1 997 



Page 9 



Questionnaires 

Questionnaires are assumed to be valid because they can be anonymous and are therefore likely 
to encourage honesty. It is recognised that the use of questionnaires is not a simple matter when 
dealing with children. The difficulty of misinterpretation and misunderstanding of questions 
was considered and a trialling of the questions was conducted with a similar sample of children 
at another site before the pilot study. It is acknowledged that because this study involved 
children, some with limited written literacy skills, the questions needed to be simple. The 
questions were trialled to increase the probability of age appropriateness and I was present 
during the implementation of all questionnaires to clarify questions. 

The disadvantages of questionnaires were not a major problem in this study. Interviews were 
conducted immediately after the implementation of the questionnaire. Issues which arose from 
the questionnaires were able to be discussed where necessary. 

Self-reporting 

Self-reporting is often associated with studies of metacognition. When a student is asked to 
self-report they may be given a hypothetical or real problem to discuss. They are asked to either 
concurrently, retrospectively or hypothetically explain the strategies they might or have applied 
to the problem. The stated strategies are used as evidence of metacognition. In this study 
students were asked to solve two problems in the questionnaire and asked to record what they 
did. The interviewed students were asked to solve another problem and to talk through their 
thinking strategies. Students were not asked to talk aloud (as in some studies, for example, 
Lesgold, Lajoie, Logan, and Eggan, (1990) as they solved the problems because it is believed 
that the verbalisation of the approach to problem solving could create the effect it sought to 
monitor (Clarke, 1992 and 1996). 

Types of tasks in the questionnaire 

Two mathematics problems were included as part of the questionnaire for students to reflect on 
their thinking processes. One further problem was given to the students who were interviewed. 
All problems came from a teacher reference on problem solving book ( Milton, 1986). They all 
had one right answer and were able to be completed within approximately five minutes. The 
problems were non-routine and chosen to engage students. They were purposefully challenging 
because it is recognised that some students do not have to reflect to solve many classroom 
mathematics problems (Fortunato, Hecht, Kehr Tittle, and Alvarez, 1991) and therefore the 
tasks would not provide an effective basis for reflection in the inventory section of the 
questionnaire. 

All of the problems had the potential to engage students in visualisation and/or diagrammatic 
representation of the problem. One questionnaire problem and the interview problem required 



4 November 1997 

•jl! 



Page 10 



students to use the operations of multiplication or addition. The problems enabled students to 
select from a variety of approaches to solve the problems such as: trial and error, process of 
elimination and working backwards. There are other ways which the tasks may have been 
categorised, for example, Williams and Clarke, (1997) but such fine distinctions are unlikely to 
contribute to the analysis in this paper. 

Inventory 

The questionnaire was based around an inventory of metacognitive behaviors. Students were 
asked to indicate whether they always, sometimes or never did these things when they solved 
problems. This technique (sometimes called Likert scales, rating scales or self-report 
questionnaire) for monitoring metacognition has been used in other studies such as those 
briefly reviewed below. In the search of education literature, no inventories were found for 
assessing the metacognition of children at the grade six level in mathematics. Most inventories 
have been completed with students at tertiary or secondary levels. 

The mathematics questionnaires used by Fortunato et al (1991) and Stacey (1990) were influential in 
the design of the initial questionnaire in this paper. Both were conducted with secondary school 
students therefore they needed to be simplified and modified for primary school age children. 
Neither Stacey or Fortunato et al used open ended questions in their questionnaires (like those used 
in this study) but they both provided the stimulus for thinking about ways to effectively tap the 
thought of student thinking. 

Fortunato, et al (1991) designed statements which focused on various stages of student thinking 
while problem solving in mathematics: planning, monitoring the process, evaluation and execution 
of the problem. They acknowledged the contributions of Corno and Mandinach (1983) and 
Schoenfeld (1985) in the creation of the questionnaire statements. Stacey (1990) relied on Likert 
scales developed by herself, by Bourke (1984) and the version of Fennema-Sherman scales as 
adapted by Rowe (1988). 

Other researchers have used this technique which is fundamentally similar to the method used in my 
research. In a study of metacognition in mathematics conducted in secondary schools, Goos ( 1995) 
used both Likert scales (based on the work of Clarke, Waywood and Stephens (1993) Me 
Donough and Clarke, (1994), and Schoenfeld (1989) and open ended problems. She acknowledged 
the work of Garofalo (1987) and Schoenfeld (1989). Grubaugh and Speaker, Jr. (1991-2) used 
rating scales with college students to discuss and self-assess their metacognitive abilities in 
reading/writing and study skills. Grubaugh and Speaker, Jr. (1991-2) used the same basic model of 
metacognition as used in this paper (ie awareness, monitoring and regulating of one’s cognitive 
processes.’ (1991-2 : 45) but the language employed in the scales is complex and specific to the 
English domain. 
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The final design and implementation of the initial questionnaire was determined by the needs and 
capabilities of the sample and the study parameters. A combination of question types were included: 
Likert scales and open ended questions. Likert scales were chosen to form a part of the 
questionnaire and interview process in this study. The items were based on the behaviors specified 
in the Metacognitive Behavior Table (see appendice 1) and an examination of the literature. Open 
ended questions were considered essential to allow for unexpected or unanticipated answers which 
may suggest unthought-of relationships or hypotheses. 

Biggs (1993) cautions about the development and analysis of inventories, reminding readers that 
factors related to the student, teacher and context cannot be ignored. This has been acknowledged in 
this study. Checks on context dependence, such as the interviews have been included. 

Pilot study purposes 

Before commencing a larger study it was deemed useful to validate the potential of the 
instruments for collecting useful data. The major purposes of the pilot study were to check 
interpretations of the questionnaire items, remove ambiguities and test the adequacy of response 
categories. Because of the complexity of the issues involved in this study, student 
understanding of the language employed in the questionnaire had to be trialled. 

Sample 

Questionnaires and interviews were implemented with two groups of children. One group were 
from an outer suburban school (A, n=8) and further questionnaires were given to an inner 
suburban school (B, n=7) group of grade six children. A total of 15 grade six children from 
two schools were given questionnaires. At school A, 5 of the 8 were interviewed immediately 
after the questionnaire implementation. None of the students from school B were interviewed. 

The teacher at school A was given a simple questionnaire. This required her to indicate which 
metacognitive behaviors from the student questionnaire list she had emphasised during 
mathematics lessons. 

Results from the student questionnaires administered at both schools were compared to see if 
general trends were possible or if results were peculiar to each setting only. The distribution of 
responses from school A was similar to that of school B. 

Copies of the student questionnaire and student interview schedules are not included in this 
paper. Many parts do not pertain to issues discussed in this paper. Relevant sections are 
reproduced in the next section. 
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Results 



This paper focuses on the analysis and interpretation of the student questionnaire and student 
interview data. An analysis of the teacher questionnaire has been completed but is only briefly 
reviewed in this paper. A more detailed analysis is reported elsewhere (Wilson, 1997). The 
findings will report on: 1. Findings about student metacognitive behavior and 2. The need to 
refine aspects of the research design and methods. The latter considers how effective the 
instruments were for gathering data on metacognition and raises questions for the main study. 

1. Results about metacognition 

Subjects have been grouped according to the similarity of their responses to particular items. 
Results of the metacognitive behavior items section of the questionnaire were represented 
diagrammatically as shown below. Where the number of responses for any cell was below n= 
3 (20%) the cell was considered empty. The shaded cells indicate how most or all students 
responded. The diagrams took the following forms: 



A. Students who said they always do this 



| Never 


| Sometimes 




B. Students who said they never do this 


l Never 1 


Sometimes | 


Always | 



C. Students who said they sometimes do this 



Never 



Always 



D. A spread of results. Where some students said they did and some said they did not. 



lever 



: T ■ t i;- ; 



' 



Sometimes 



Items which fell within types A, B and C were not considered useful because they did not 
discriminate between students. Type D showed a division between responses and that was 
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considered interesting. It raises the question: What do the people in each group have in 
common? 

NB. The metacognitive behaviors have been coded. This appears in the brackets next to each 
item by function as defined in the metacognitive model (Figure 1). The items have been 
included here as they appeared in the questionnaire. 

Category D: The items which show a range of responses and therefore differentiate between 
students include: 

b. Ask yourself a question about the problem, (awareness, evaluation and/or regulation)* 

g. Think about what you'll do next, (regulation) 

h. Go back and check your work, (regulation) 

*The type of question students may be asking themselves may be very significant and needed 
to be followed up in the interview. For example. Are students asking a question about the 
requirements, what they know, their own abilities, how they could change their strategies? etc. 

Using this classification scheme students (Category A) students responded that they always: 
a. Think about what you already know, (awareness) 

c. Think about what the problem is asking you to do. (awareness) 

d. Make a plan to work it out. (regulation) 

e. Try to remember if you have ever done a problem like this before, (awareness) 

There were no items which fell within type B. 

Students said they sometimes: 

f. Guess the answer because it like a problem you've done before, (regulation) 

i. Think about a different way to solve the problem, (regulation) 

j. Change the way you are working.** (regulation) 

**It is recognised that the response to this item could be misleading because you may on most 
problems, only sometimes change the way you're working. If you were metacognitive you 
would only change the way you were working if you needed to. Indeed all behaviors listed on 
the questionnaire could be present during problem solving sometimes. It is noted that all items 
of Category C type were regulation functions. 

People often suggest that student reports of their behaviors are merely a recital of teacher's 
communicated values. In this study there was little correspondence between what the teacher 
said she did (from the metacognitive items list given to students) when instructing the class and 
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what the students reported they did in the questionnaire. In the questionnaire the teacher noted 
what she emphasised during mathematics lessons. This did not relate directly to student 
responses. Interestingly, on category A items (where students demonstrated a range of 
responses) the teacher said she never instructed her students to ask questions about the problem 
(item b), that she sometimes asked them to think about what they would do next (item g) and 
that she always asked students to go back and check their work (item h). All items and 
categories were analysed in a similar way to compare the student and teacher responses. No 
clear connection was made. 

Metacognition: The connection between awareness, evaluation and regulation 

The model of metacognition as defined in this paper has been checked for consistency with the 
results of this study. While-the data analysis results are consistent with the hypothesised 
sequence suggested, evaluation of the data collection methods used concludes that the 
development of further instruments is required to confirm the link between the metacognitive 
functions of awareness, evaluation and regulation. For example, Do these functions need to 
always happen in this sequence? Does the data prove that students do not regulate unless they 
are aware of their thinking and evaluate their thinking? 

2. Research design and methods 

The pilot study revealed a need to change some aspects of the student questionnaire and 
interviews. Aspects of the design needing fine tuning and reconsideration are listed below. 

a. Changes to the questionnaire 

Language and Items: At a minor change level, results from the study demonstrated a need to 
simplify the language and reformulate some questions. It appeared that some questions also 
needed to be more specific. For example, Question (b) asked students if they ‘Ask themselves 
a question about the problem.’ Responses did not indicate which specific metacognitive 
function was in use. This question could include examples about what question students might 
be asking themselves, such as: about their thinking or about the way to solve the problem. 

The questionnaire items should have asked students to reflect on the task they had just done 
(current metacognition) and what they usually do when they solve mathematics problems 
(retrospective reflection). Of more importance the analysis of the questionnaire data 
demonstrated a need to add more items to the questionnaire. An uneven number of items related 
to the different functions of metacognitive thinking behaviors was found. More evaluation 
items were needed. 

Tasks: Pilot questions were trialled prior to the implementation of the pilot study. During 
trialling of the intended pilot questions, a problem was encountered when children were asked 
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to recall decisions they made during mathematics problem solving. When routine mathematics 
equations were chosen by children to represent mathematics problem solving they did not 
reveal metacognitive thinking. To correct this problem, the final draft of the pilot questionnaire 
included non-routine problems for children to solve. This created other difficulties such as, the 
choice of appropriate problems, what number of problems should be included, the fact that the 
problem was out of context etc. In order to increase the chances of getting responses which 
would illuminate metacognitive thinking, the decision was made to include non-routine 
problems, creating a common reference point and context for research subjects. 

Although the use of non-routine problems was more effective in stimulating and reporting 
metacognitive thinking than the use of the use of routine problems, three different types of 
problems will be used in the main study. These will include one spatial task, one task involving 
calculation and another task requiring students to use logic. 

Other strategies cited by students 

Some students responded to the last open ended question in the questionnaire which invited 
them to add anything else they do when they solve mathematics problems. This question was 
included to check that all possible responses were allowed for. Most of the strategies listed by 
the subjects were low level strategies, many were about how to get started or not specific 
enough to be illuminating. One student said “think logically and carefully.” Unless specified 
the responses were stated between 2-3 times. The responses included: 

• Read the question 

• Write it down 

• Use gimmicks 

• Asks others (4 responses) 

• Use visual aids/representation, for eg, diagram, fingers, pictures in the mind. (Most 
frequently stated, 6 responses) 

Many of these strategies listed by students may have been consciously used as a result of their 
Awareness or Evaluation of their thinking or to change their way of working (Regulation). 
Students may have done these things as part of their ‘usual ways of working’ or as suggested 
by their teacher. This possible interpretation of the questionnaire data would be need to be 
verified through student interview. Further research would need to consider whether some 
items on the questionnaire need to include aspects of these student responses. 

b. Student Interviews 

The interviews provided the opportunity to follow up discrepancies between individual 
questionnaire responses on different tasks and for students to clarify questionnaire responses. 
The interviews also provided the opportunity for the researcher to observe students as they 
solved another problem and to record student comments about the strategies they used. 
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The interviews were disappointing as they were not very revealing in terms of additional data 
about metacognitive thinking, beyond that already provided by the questionnaire. Many student 
responses highlighted the difficulty students had reporting on their own thinking. For example, 
when asked about the strategies they used when solving mathematics problems many students 
responded as such: “Don’t know”, “It came automatic”, ‘It just hit me”, ‘Just known before’ 
and “Did it in my head”. Sometimes these response were followed by comments offering 
insight into student thinking. For example, the following two quotes suggest that regulation of 
thinking may be related to the perceived importance of the task. These responses followed a 
question about whether students assessed themselves in mathematics. 

“Yes, occasionally I think about how I could have done it better. ...When I 
understand it's easy. I think about my answers and if it's good and correct. If I 
could do it a better way. Try different ways. Even try them, sometimes. If I think 
it's a better way to do maths questions and you get better marks I'd probably write 
it out better-if it's like a test. If it's like normal classroom maths, I always try hard 
but I put more effort into a test." Student A 

"Don't know. When I do maths I just think about the question- only if it's a test 
because that's really important. I think I'm getting it wrong and then I suiprise 
myself. How could I improve? I don't like practising." Student B 

Tests seemed to provide the motivation for encouraging action. Harris and Bell (1986) and 
Tang (1994) support this notion. They argue that students’ awareness of assessment and 
perceptions of the assessment demands can effect their attitude and learning approaches. 



Discussion 



The findings of the pilot study raise many questions about the assessment of metacognition. 
They have acted as a vehicle to draw conclusions about research design and have implications 
for the main study. In this study (and others) defining metacognition and relating this 
definition to methods for reliable data collection for detecting metacognition has presented a 
methodological challenge. These difficulties and limitations have been discussed within this 
paper. 

Analysis of the pilot study supplemented by additional readings of the research literature 
suggest particular key findings and associated action which are required for the design a reliable 
and valid study to assess metacognition. The proposed actions respond to the objections and 
findings, and provide direction for further research. 
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Key Findings and Action 
Key Finding One 

Existing definitions of metacognition are insufficiently precise and difficult to interpret in 
operational terms. 

Action One 

Refine the Metacognitive Behavior Table and inventory items to use to assess student 
metacognitive behavior. 

Key Finding Two 

Self-reporting appears to be a valid method for researching metacognition when used in 
conjunction with other reliable methods and checking procedures. 

Action Two 

Design the data collection to minimise minimum time between the questionnaire and interviews. 
Corroborating evidence should be provided to substantiate and elaborate questionnaire 
responses. For example, through an interview and/or others’ observations by the teacher 
and/or a researcher. To realise this goal, the teachers would need to be provided with a 
definition of metacognition and asked to rate their students' metacognitive and mathematics 
abilities. 

Conclusion 

The model of metacognition and inventory as reported in this paper appeal - to be viable and 
effective in guiding both data collection and analysis but new data collection instruments need 
to be considered. An assessment strategy for metacognition would be more attractive to 
teachers if it could be used as an integral part of instruction. The design of such a useful 
instrument is a central goal of the research project of which this study was the first stage. 
Further work is currently being undertaken to this end. 

In this paper it has been argued that assessment practices must move beyond the basics. If 
assessment of students' metacognitive thinking is not integral to teaching and learning, the 
improvement of student metacognition cannot be assumed. Only a basic curriculum and 
education for our students can be expected. 
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Appendice 1. Metacognitive Behavior 



This table shows what students may be thinking about when they are reflecting on their 
thinking as well as indicators of metacognitive behavior. 

1. Monitors 

When children monitor their thinking (awareness and/or evaluation) they ask questions, self- 
assess and make links between ideas. 



Metacognitive Thinking Metacognitive Behavior 

a. Monitoring Self 


• own thinking about: 
learning 


Self-assesses learning 
Questions learning. 


own abilities 


Self-assesses own abilities 
Questions own abilities 


own progress 


Self-assesses own progress, 
Questions own progress 


own thinking 


Self-assesses own thinking 
Questions own thinking 
Links current and previous ideas 


own behavior 


£elf-assesses own behavior. 
Questions own behavior 



b. Monitoring Practice 


•tool use 


Asks a question about requirements 
Relates to another idea/task 
Makes prediction based on another idea/task 
Lists other ways to approach task 


•choice of strategy 


Self-assesses choice of strategy 

Asks a question about the choice of strategy 


•own use of strategy 


Self-assesses their own use of strategy 

Asks a question about their own use of strategy 



2. Regulates 

When children regulate their thinking they make lists, decisions, self-correct, select and change 
their ways of working, set goals and plan action. 



Metacognitive thinking 
a. Regulating Self 


Observable Behavior 


•learning 


Self corrects own learning 


•thinking 


Self corrects own thinking 


•behavior 


Self corrects own behavior 




Makes a plan 




Sets goals 



b. Regulating practice 




•strategy use 


| Selects a new strategy 
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